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ABSTRACT 

Designed to complement the manual, "Evaluation o£ 
Educational Software: A Guide to Guides," which provides an overview 
of 10 different models of microcomputer courseware evaluation 
instruments, this 5-section workshop guide outlines an approach to 
using the evaluation materials in a hands-on workshop setting. 
Session notes, reproducible handouts for the participants, and 
suggestions for implementation are included. The workshops begin with 
a discussion of key features of instructional and message design, 
including learning (cognitive) processes such as use of aids to 
memory and attention; language use and text characteristics; graphics 
and visual processing; the user's cognitive model; and feedback 
techniques. An overview covers the range of forms available in the 
"Guide to Guides." Procedures for hands-on evaluation of a correlated 
set of software examples (not included) are suggested, as well as 
alternatives to using the examples for groups without access to a 
compatible computer. The next section provides opportunities to 
describe and compare procedures, criteria, and formats of the 10 
evaluation forms from the "Guide to Guides." Notes for the final 
session describe evaluation resources. Additional suggestions for the 
workshop group leader are included. (LMM) 
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PREFACE 



SUGGESTIONS FOR SESSION LEADERS: 

The SEDL Workshop based on the Evaluation of Software: A Guide to 
Guides is intended for educators who are involved in the selection of software 
for use with educational computers. The materials contained in the Guide to 
Guides were compiled in order to produce a resource book devoted to software 
evaluation. They are therefore oriented toward instructional resources used 
with educational computers. The Guide includes awareness and reference mate- 
rials on informed software selection, ten different models of evaluation 
instruments, and information on resources which might prove useful in esta- 
blishing a system of software evaluation. 

The Bx Workshop is divided into five major sections, plus an evaluation- 
instrument which is intended to measure the participants' responses to the 
module. The workshop itself consists of the Session Notes, to be used by a 
group leader or trainer as a guide during the workshop, along with Participant 
Handouts correlated to sections of those Notes; a copy of the Evaluation of 
Educational Software; A Guide to Guides ; and a sample disk copy of software 
programs for use in the hands-on evaluation portion of the training. The disk 
contains a range of educational programs, with varying content and grade 
level, and is compatible with a TRS-80, Model III computer. 

Section One of the Notes is designed to characterize some key features 
of instructional and message design which may be important in effective soft- 
ware. The five areas covered in Section One include learning (cognitive) 
processes such as use of aids to memory and attention; language use and char- 
acteristics of text presented in the program; use of graphics and visual pro- 
cessing; the cognitive model of the user; and feedback techniques (Jay, 
1983). The Session Notes in Section One are intended for use as a framework 
for group discussion on the points presented. The participants will be pro- 
vided with handouts for each area in this section, outlining the major 
points. The handouts are to be used as a means of recording comnents and 
individual^ concerns regarding the applicability of the concepts to each 
person's situation. 



In Section Two, the participants will be given an overview of the range 
of forms available In the Guide to Guides , Ten different evaluation forms are 
included In order to Introduce participants to a variety of format and con- 
tent. The group leader should take note and remind the participants that the 
forms In Guide to Guides are only a representation of the field of evaluation 
instruments available at time of publication (as of Spring, 1983). These 
forms do not necessarily provide a definitive set of evaluation procedures, 
but they do represent some systems which have proven to be effective under 
given circumstances. 

Evaluation of appropriate materials for i s o with microcomputers consists 
of more than sirrple application of a form. The exercises in this section, 
which exposes the participants to various instruments of evaluation, repre- 
sent a step in the total evaluation process. The participant handout in 
Section Two is intended for consensus work in a small group session but rnay be 
effectively used by an individual. The participant (s) are to decide which 
elements listed are important in their review process, and which of the ten 
forms included in the Guide to Gu ices most satisfactorily account for those 
el ements. 

Participant Handout 2-A is a comment form, also for small group con- 
sensus activity, to be used in choosing several of tha forms which appear to 
be appropriate for the hands-on evaluation section. The tr^'ner is advised to 
inform all participants that this system of selecting severoil of the forms for 
the hands-on evaluation session is merely for the convenience of time-saving 
during training. 

IMPORTANT! 

No one form should be eliminated or deemed not useful on the basis of the 
checklist activity. Each form should be considered as an alternative in 
specific cases. It is possible that an instruct'i onally sound program will 
perform poorly on one for and rate highly on a different evaluation instru- 
ment. It should be stressed that this training is meant to expose partici- 
pants to the techniques of software evaluation and the types of forms in Guide 
to Guides . It is not intended to in .ate the "best'' form or to determine the 
instructional soundness of the software samples included. 
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Section Three provides for hands-on evaluation of the examples of soft- 
ware includricl in the packet. Alternatives to using the packet's software sam- 
ple are described for those groups which do not have access to computers wnich 
are compatible with the enclosed software. The group leader 1s advised to 
become familiar with the hardware used in the evaluation session. He/she 
should be able' to load programs and instruct others in the basic procedures 
for using the programs being evaluated. During this section of the training, 
participants will have an opportunity to discover elements of software design 
which could be identified as appropriate to the educational settings for which 
they would be evaluating software. 

Section Four provides an opportunity for participants to describe and 
compare procedures, criteria, and formats of evaluation forms from the Guide 
to Guides . The comparison exercise is intended for a large group discussion, 
facilitated by the leader. Specific suggestions for leading the participants* 
comments in constructive ways and documentation procedures for those comments 
(Participant Handout #4) are included in the Session Notes, 

Section Five describes print and other resources which deal with educa- 
tional software evaluation and reviews. The group leader is advised to 
attempt to secure copies of some of the publications listed in the resource 
section of the Guide to Guides , so that participants can be exposed to differ- 
ent types of information sources. 

EVALUATION OF THE TRAINING SESSION: 

At the end of the training materials in the module, the participant will 
find an evaluation form which refers to the effectiveness of the training ses- 
sion. The group leader should direct each participant to fill out the form as 
completely as possible. Evaluation on each training session using the 
Workshop will be compiled by the SEDL Regional Exchange staff and used to 
improve future training materials. 
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Any questions or concerns regarding the SEDL Workshop or the 

Evaluation of Software; A Guide to Gui des should be directed to.* 



Linda A. Lloyd 
Dissemination Specialist 
Regional Exchange 

Southwest Educational Development Laboratory 
211 East 7th Street 
Austin, Texas 78701 
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[•VALUATION OF EDUCATIONAL SOFTWARB; 



Ratlonaje i 

According to a recQnt survey concluctecl by Markot Data Retrieval, 24,646 
schools are using computers In an Instructional capacity, a 565(1 Increaso ovor 
1981 figures. Another source cites 60% of school districts nationwide, and 
31% of all schools as having access to microcomputers In 1982. The Increase 
In educational computer placement In public schools across the country In 1983 
Is probably going to be Inevitable. 

The Integration of the microcomputer, and the technological training and adap- 
tation requirements which will accompany it, may pose vast problems for educa- 
tors. A critical area of concern for school administrators, state departments 
of education, and local education personnel is the evaluation of educational 
software. Educators who find themselves in the position of selecting and 
reviewing software are suddenly responsible for evaluating a number of factors 
relating to the Instructional soundness and applicability of curricular mate- 
rials that may not fit well into a standard courseware evaluation format. The 
task of choosing software can include determining the material 's appropriate- 
ness for grade level and subject area, soundness of instructional design, and 
placement in curricular scope and sequence across the grade levels. Some sim- 
ilarities exist between software evaluation techniques and more traditional 
educational evaluation of materials, such as textbooks, worktexts, and other 
mediated materials (including audio tapes, fllmstrips, films, etc.). However, 
the evaluation of educational software presents unique challenges. 

Reviews in educationally oriented microcomputer journals vary widely in relia- 
bility, primarily because of differing rationales and standards for review 
selections. The reader must determine whether the reviewer is discussing the 
material on a level that relates to the classroom or situation for which the 
software in question is being considered. This is a classic '*apples versus 
oranges" comparison in which one must be certain that the intent and orienta- 
tion of the author of the review is similar to that of the reader/consumer. 
One of the most difficult tasks for the reviewer of software is determining 
appropriateness for a given set of students. Often the assigned evaluator for 



a school or distriql; has no prior experience with the r^^nge of softwdre AvaiN 
(ibl'--, find nmst face <\ staggering variety of educational materials. Some 
schools and districts have adopted a coiniilttee approach to software screen- 
ing. But most cHstrlcts are on their own when evaluating Instructional prO'« 
grams for tho schools' nilcirocomputers. 

As a means of addressing the Issue of software evaluation, the Evaluation of 
Ediicationa] Software; A Guide to Guides (1903) was producQd by the Southwest 
Educational Developnient Laboratory and Northeast Regional Exchanp, Inc, This 
manual 1s a collection of ten different evaluation forms, along with sugges- 
tions for software evaluation and lists of resources useful to those making 
Informed software decisions for educators. 

This Jiy^ Workshop has been developed by the Regional Exchange at SEDL and 
provides practical training for all educators who plan to use the Guide to 
Guides as a tool for responsible software evaluation. 



• to provide an overview and facilitator/trainer's package, so that 
administrators from state education agencies will be able to dissemi- 
nate training Information on evaluation of software to local and dis- 
trict level educators; 

• to provide awareness of the ten models of software evaluation repro- 
duced in Evaluation of Educational Software; A Guide to Guides , and 
to give' hands-on experience in using two or more of these evaluation 
forms with actual software; 

• to provide expanded information on print and organizational resources 
that are useful in making software decisions. 



Goal s: 



The 



Workshop has three overall goals: 



-vn- 



13 



ERIC 



Objectives; 



The workshop packet has been designed to accomplish certain specific objec- 
tives. By the end of the training exercise the participants will be able to: 

• define or identify key issues in the software selection process as 
those issues relate to the participant's educational situation 
(Rationale); 

• characterize the general features of instructional design as applied 
to software evaluation (Section 1); 

• describe and compare procedures, criteria, and formats of evaluation 
forms contained in Evaluation of Educational Software: A Guide to 
Guides (Section 2) ; 

» discover specific elements of "good" software for their given situa- 
tions through hands-on evaluation of software (Section 3); 

• describe or identify print and organizational sources of information 
on software evaluation, reviews and databases (Section 4). 
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INSTTIUCTIONAL DESIGN CONSIDERATIONS IN EDUCATIONAL SOFTWARE 

Most educationally-related software materials are designed with some common 
information-processing abilities in mind. The evaluator may wish to account 
for some or all of the following processes when choosing appropriate software. 

Memory and Attention: (corresponds to Participant Handout #1-A) 

Evidence from years of psychological testing and experimentation sug- 
gests that human beings can only deal with a certain amount of informa- 
tion at one time. The level of difficulty and the number of tasks 
required may determine the span of attention. 

PRINCIPLE: Short-term memory is limited by space and time. 

PRACTICE: Implement programs which: 

1) Present one idea at a time: do not fill a screen with 
solid text. An idea In text is probably limited to one 
or two sentences in length. 

2) Consider timing: processing text material takes time. 
Give the user control of the pace of instruction ("Press 
any key to Continue"). Timing may be self-paced and 
fl exibl e. 

3) Keep the user informed: in the event that the program 
has long delays for searching or other processing func- 
tions, watch for programs that explain the waits. A 
prompt on the screen, such as "I am searching" or "Wait, 
please, while I find that answer," might appear as an 
explanation for a delay* A blank screen for more than 
two or three seconds may be interpreted by the user as 
"trouble" with the machine. 
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4) Consider the level of difficulty: the more complex the 
idea presented, the more time is needed for inspection. 

5) Use some type of supplemental materials: because some 
students often have trouble retaining a number of ideas 
at once, often they will perform better with printed 
reference- materials. These may include worksheets, 
maps, tables, equations, diagrams, instructions or 
memory joggers. Materials may be either teacher's guide 
supplements, materials from work texts or sheets 
included in the program package, or teacher-made mate- 
rials which support ideas in the program (this entails 
the teacher's advance preparation). 

6) Allow for differences in level of processing: the more 
deeply the student is required to think about a subject, 
the better overall retention will be. Simple assess- 
ment of an idea ("Is the word "dog" in upper or lower 
case letters?") is less likely to produce retention of 
the word do£ than a more complex task. A task/question 
such as "A poodle is a kind of ", with an inter- 
active response relating to the word dog , is more likely 
to produce retention in the long-term memory mode. 
Generally, the more cognitively elaborate the process, 
the better long-term memory results. 

7) Have a reasonable duration of program: the greater the 
mental (processing) demands of the task, the shorter 
should be the duration in order to keep performance at a 
satisfactory level. In all programs, opportunities to 
exit without penalty should be provided. 

Language or Text Characteristics: (corresponds to Participant Handout 
#1-B) 

The way messages, instructions, and text in a program are phrased can 
affect student performance differentially. Simply stated, attention 
must be paid to the content of the message (style, syntax, vocabulary), 



the type of user (adult, child, special -needs learner); and the intent 
of the message (to provide information, pose questions, or prompt cer- 
tain behaviors ). 

PRINCIPLE: The performance of the student is a function of the type of 
message , the intent of the message , and the type of student, 

PRACTICE: Supplement programs which communicate effectively with your 
audience by using some of the following ideas: 

1) Vocabulary: new words, terminology, or jargon need to 
be defined and explained. Jargon should be avoided, if 
possible, unless the intended audience is definitely 
familiar with words and references used. Keep in mind 
that children do not always understand words in context 
the way adults may. 

2) Humor: be careful of the type and level of humor in the 
program. What Is funny to one group or person may mean 
nothing to or offend another. Sarcastic, off-hand com- 
ments inserted Into programs as "comic relief" may be 
misinterpreted by some learners as criticisms. In gen- 
eral, some humor can Improve the effect of a program. 
Keep In mind that, although often appealing to students, 
humor does not have any consistent beneficial effect on 
acquisition of Information. 
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3) Natural communications: check to see if programs 
respond to and cotmiunicate with the student in a "user- 
friendly" manner. Users may attribute a personality to 
the machine and software. Many responsive programs 
refer to the student in the first person voice and call 
the student by name. Some teachers may wish to point 
out that the machine is not a person, but merely is 
responding to students in a personal manner to encourage 
answers and responses. Many programs use "I", "we", or 
"you" when directing or correcting the student. 
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4) Avoid "codes": programs which request an answer of 
"yes" or "no" should respond to "yes" or "no" rather 
than Y or N (or even worse, 1 or 0). If a program uses 
coded responses, look for those which document the fact 
that coded answers are going to be used and which are 
consistent in the practice throughout the content 
course. 

5) Be con si stent i n presentation and student response 
modes within a program: again, consistency of format 
and expectations can significantly reduce errors by stu- 
dents confused by arbitrary changes. Consistency 
reduces errors and frustration. Do not swi::ch codes or 
answer formats unless warning the user and explaining 
the reason. 

6) Concrete versus Abstract: language that is graphic and 
grammatically simple is usually better remembered than 
abstract wording. Images (graphics) are better recalled 
than words. Younger children usually think better in 
concrete (real) rather than abstract (imaginary) terms. 

7) Format of presentation: memory may be enhanced by mul- 
tiple presentation formats. A strong basic presentation 
incl udes information presentation via more than one 
mode, if possible. Graphic, print, and auditory chan- 
nels are all accessible on the microcomputer, and when 
appropriate, can improve reception and retenti on of a 
concept. While this may seem to contradict the limited 
capacity notion in actual practice, redundant presenta- 
tion of a concept can enhance retention and performance, 

8) Sentence format: generally, look for simple, concise 
language. Pay attention to word usage, sentence struc- 
ture, and sentence length. Remember to present one idea 
at a time on the screen. Watch spacing between lines 
and words. Don't allow graphics to overwhelm printed 

ideas or directions visually. 
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9) Intentions of the message: memory for concepts can be 
improved by making the receiver of your message think or 
process information. Presenting simple material may be 
accomplished by basic dril 1 - and-practice . If improve- 
ment of reasoning skills or problem solving abilities is 
the desired skill acquisition area, the in quiry method 
may be more useful. A dialog between the user and the 
software will require reasoning skills and result in 
more abstract learning. Both abstract reasoning skills 
and the knowledge base which is reinforced by 
dri 11 /prcjctice. type settings are valuable tools for 
classroom learning. 

Graphics and Visual Processing: (corresponds to Particip.^nt H^jndcut 
#1-C) 

Learners process information with all the senses. Vision is generally 
the dominant sense and the major input channel from the con)f>uter. Soft- 
ware in the future may include more graphics and color, and more anima- 
tion of graphics and characters as equipment becomes more sophisticated. 

PRINCiPLE: Graphics may be used to enhance retention and to encourage 
attention to task. 

PRACTICE: Implement programs in which color is used to: 

1) color code, differentiate, or highlight instructional 
sequences. Color may enhance the realism of graphic 
representations. It also directs attention to signifi- 
cant aspects of the display* 

2) Use color cueing to direct attention to signi ficant 
features and ideas on the screen. Color may be used to 
focus the student's attention on certain visual aspects 
of the stimulus. This feature might be particularly 
useful in teaching distinguishing features of a subject, 
or analogies and comparisons. 
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3) Graphic descriptions or anaV?nies are helpful in 
describing abstract ideas. Students told to imagine a 
concept in light of a concrete example learn and retain 
such information better (i.e. , imagining chemical 
structures as "tinker-toy" images). Such "imaging" 
techniques may help learners to remember concepts better 
than using only abstract verbal instruction. 

4) Pictures, maps, diagrams and drawings^ may be used in 
conjunction with computer i nst ruction. Some graphic 
displays may be unworkable via computer. Supplements 
are cheaper, lessen memory demands (both computer's and 
learner's), and in general, enhance learning. 

5) Graphics may be used to emphasize important informa- 
tion. Special characters, underlining, blinking words, 
enlarged text and color are all elements which may be 
used as graphic emphasizers. 

6) Mnemonic skills can stimulate retention. Word meanings 
are facil itated by imagining or visual i zing the con- 
cept. Graphics which encourage students to use visual 
memory as a prompt can improve the effectiveness of a 
program. 

7) Color use should agree with realistic denotations. Red 
means stop, green means go, etc. Confusion can occur if 
the program uses color inappropriately in situational 
graphics requiring common color relationships for credii- 
bility. The wrong use of color may detract from the 
effectiveness of message delivery. 

8) Use color to teach relationships and differences in a 
displ ay. Color should be used to denote similar 
functions, properties and characteristics of a concept. 
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Cognitive Model of the User: (corresponds to Handout #1-D) 



The profile of the user or student is a collection of real and assumed 
abilities that the evaluator of software must formulate. It is this 
image of the learner and his/her capabilities that must be considered in 
determining appropriateness of courseware. Nearly all teachers have 
developed implicit models of their students and choose educational mate- 
rials that fit into the perceived learning styles of the students. Stu- 
dents are instructed based upon these real or implied abilities which 
are assumed by the teacher/trainer. One of the most important functions 
of the computer is its ability to teach at each individual's level of 
comprehension. Some of the learner abilities that should be considered 
as variables in software selection evolve from the following principles: 



PRINCIPLE: Processing abilities mature with the age of the learner. 

Researchers such as Piaget detail the levels of processing 
in their studies. The devel opmental reasoning growth of 
children and the differences in their abilities to think 
abstractly, hypothetically , deductively or logically are 
areas to incorporate in a learner's profile. These abili- 
ties usually develop in stages according to developmental 
age. The student must be taught at his/her level of 
reasoning. 



All learners have individual learning styles but special 
education students have specific instructional needs. 
Gi f ted students are prone to boredom when presented wi th 
simp 11 Stic programming. Program courseware desi gn for 
handicapped learners requires more structure and would 
ideally include a multimedia approach. 

PRACTICE: Implement programs which are appropriate to the learner's 
cogni ti ve styl e: 



1) Entry level: It is important to consider the learner's 
pri or experience or instruction on the topic of the 
cothrseware being evaluated. The current knowledge level 
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of the prospective student should determine when or i f a 
piece of software is appropriate. Materials with too 
many unknowns (concepts not previously taught or 
encountered) can be frustrating if the program does not 
allow for the naive learner (via branching or reinforc- 
ing prompts). 



2) Graphic presentation of a concept is an excellent rr^eans 
of emphasizing an idea for any age learner. Enlarged 
text is a type of graphics utility which may be used 
with young or visually i mpai red students. When pos- 
sible, auditory or voice capability may be used to 
benefit students who learn best through multiple-presen- 
tation techniques. 

3) Concrete reasoning can be expected in children under the 
age of eight, or with students who are academically 
handi'.apped and functioning at a lowered developmental 
level. Use descriptions and images that can be directly 

- compared and applied. Agai n, multi media presentation 
can enhance performance. 



4) Teen-agers and bright upper elementary students can han- 
dl e hypothetical situations, deducti ve reasoning, and 
problem-solving tasks at an adult or nearly-adult level. 

5) Courseware that can encourage further exploration of the 
topic presented is useful. Play and experimentation 
with ideas, as well as concise statements of fact, can 
be used as reinforces to learners. 



6) An interactive approach to learning (using dialog and 
questions) can encourage learners to generalize and thus 
better retain information. Programs can chal 1 enge a 
student by separating given (entry-level) knowledge from 
new information. 
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7) Some students need more "external strategies" such as 
printed pronpts, beeps or other auditory cues, written 
directions, or previous orientation to a program than 
others, -Determine whether the audience for the software 
being reviewed needs this sort of cuing and be alert for^ 
materials which might fit that requirement. It is pos- 
si^ble in most cases to create teacher/trainer developed 
cues for programs lacking them. 

Feedback: Responding Intelligently to the User (corresponds to Partici- 
pant Handout #1"E) 

PRINCIPLE: Use of feedback to monitor performance and behavior is a key 
element in successful software. Knowing the consequences of 
behaviors upon student performance is useful to most 
learners. Performance can be improved with critical infor- 
mation about responses. Correct behaviors can be repeated 
and refined, and errors reduced, 

PRACTICE: Implement programs which provide appropriate response to the 
user: 



1) Provide information for the user when performance is 
being scored or evaluated. Provide a score at the end 
of an instructional module and within the module when 
appropriate, 

2) Keep the user informed: see Section 1, #3, 

3) Rewards for correct response should be natural and 
varied. Avoid programs where a response is used 
repeatedly and exclusi vely as a reward for correct 
behavior. Printed verbal rewards should be random and 
varied throughout the module if possible. On the other 
hand, responses to incorrect answers should not be so 
interesting or exciting as to encourage experimentation 
with wrong answers, 
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4) Hints, or "shaper" responses, may be used to lead the 
user to correct responses without telling the answers 
outright. For example, when the question is, "What is 
5x4?," and the student responds, "9," the computer might 
tell him/her that ''You added," or ''Try again, but check 
the operation sign first!" This is a more useful 
approach than a simple "wrong" or provision of the cor- 
rect answer with no elaboration. 

5) Allow for multiple answers to a question. Variant 
answers (such as spellings, sentence construction, Y or 
N instead of Yes or No, etc.) can be significant in 
many situations , and it is i mportant that the software 
be capable of accepting some amount of deviation. 

6) Personal responses I, you, we, or students' first 
names — can add to the effectiveness of prograimiing for 
some groups. It is not a highly motivating aspect for 
some audiences, however. Also, the subject matter of 
the courseware may not lend itself to informalities such 
as a dialog on a first-name basis. 

7) To avoid frustration, it is necessary for students to 
understand the processes of error statements and correc- 
tion procedures. The evaluator needs to make sure that 
students understand how to correct or revise an answer 
by backspacing and editing answers. Watch for poorly 
worded or ambiguous questions in the text which might 
encourage incorrect behaviors and responses. 

8) Competition can be stimulating and rewarding. Many 
students (extra verted, fiel d independent, and formal 
operational) respond well to trying to better their own 
performances as well as conpeting with others toward a 
common goal. More introverted students (field 
dependent) may not perform as well, in many cases. 
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9) Choose responses to incorrect answers with care. Pro- 
grams should not punish for incorrect answers, nor 
should the computer *s response simply provide the cor- 
rect answer. First, some attempt should be made to lead 
the learner to discover a correct response (see "Hints" 
#4 thi s section ). 

10) Inform the student of the initial skills (entry level) 
that he/she needs to operate a program. Some programs 
may be instructional ly sound, without overt directions 
for use or entry level definition. This is not neces- 
sary a facet of effective software. 

Recommendation s: 

Currently, many problems exist in the development and evaluation of educa- 
tional software. Some of the difficulty in responsible selection of course- 
ware may be eliminated via evaluator training activities and a knowledge of 
options in procedures for evaluation. Procedures which might be implemented 
to alleviate problems include: 

• training teachers in learning style and human psychology of 
learning, including such factors as development, cognition, 
questioning modes, perception, motivation and language; 

• conducting research on some of the factors mentioned in this 
training session; 

• extensive field-testing of educational materials; 

• reading full courseware reviews and applying skills and 
information contained in the training module. 

• matching individual learning styles and ability levels with 
particular types of software programming; 

• incorporating instructional tactics to compliment the use of 
screen,, keyboard, printer (and other peripheral devices). 
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PfllTICIPANT HANDOUTS 
Section 1 
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Handout #1-A 
Participant Notes 

MEMORY AND ATTENTION: List concerns in these areas which might relate to your 

situation. 

PRINCIPLE: Short term memory is limited by space and time. 
PRACTICE: Inplement programs which: 

K Present one idea at a time: 



2. Consider timing: 



3. Keep the user informed: 



4, Consider level of difficulty: 



5. Use of supplemental materials: 



6. Allow for level of processing: 



1 . Are of appropriate duration: 
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Handout #1-8 
Participant Notes 

LANGUAGE OR TEXT CHARACTERISTICS: Using this checklist, mention some concerns 

you hf?ve have regarding these 
characteristics of programs, 

PRINCIPLE: Performance is based upon the type of message, the intent of the 
message, and the type of student. 

PRACTICE: To communicate effectively with the student using the computer, 
implementation of some/all of these ideas may be useful: 

1. Vocabulary: 

2. Humor: 

3. Natural mode of communication: 

4. Avoid "codes": 

5. Be consistent: 

6* Concrete versus abstract: 

?• Format of presentation: 

8. Sentence format: 

9. Intentions of the message:. 
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Handout #1-C 
Participant Notes 

GRAPHICS AND VISUAL PROCESSING: List any important factors relating to 

graphics capabilities of the software, 

PRINCIPLE: Graphics may be used to enhance retention and to encourage 
attention. 

PRACTICE: Use color to: 

1. Code, differentiate, and highlight: 

2. Direct attention: 

3. Describe abstract concepts: 

4* Explain concepts and illustrate: 

5, Emphasize important information: 

6, Display mnemonics: 

7, Illustrate natural ly: 

8, Teach relationships: 




Handout #1-D 
Participant Notes 



COGNITIVE MODEL OF USER: Use the list below to pinpoint and discuss 

characteristics of learners who will be using the 
software you review, 

PRINCIPLE: Educators must consider the learner^s processing level (determined 
by developmental age), especially in instances where the audience 
is educationally a special needs group (gifted, handicapped). 

PRACTICE: 

!• Entry level: 



2. Graphics appropriate to learner: 

3. Age for concrete/abstract reasoning: 

4. Teen-agers* skill s: 

5. Exploration of ideas: 

6. Interactive approach to instruction: 



7, External cues: 
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Handout #1-E 
Participant Notes 



FEEDBACK or RESPONDING INTELLIGENTLY TO THE USER: Use the following list to 

di scuss characteri sties of 
learners, and appropriate 
feedback techniques. 



PRINCIPLE: The use of feedback as a monitoring device for performance and 
learning behavior is a key element in successful software, 

PRACTICE: 



!• Scoring and evaluation: 



2. Keep the user informed: see Section 1, #3: 



3. Vary rewards: 



4. Hints: 



5, Allow for multiple answers: 



6. Personal responses: 



7. Errors and corrections: 



8* Conpetition: 



9. Incorrect answers: 



10. Expectations: 
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SESSION NOTES 
SECTION 2 



OVERVIEW OF FORMS 



The forms Included In the Guide to Guides have been developed by various 
educational groups and organizations over the past few years. The forms are 
organized by their overall approaches to software evaluation; 

1. Evaluation strategies which are designed to be used with large 
groups of cooperating evaluators , operating under an organized net- 
work. These forms also function best with a given set of guide- 
lines: 



• MicroSIFT: 



Developed by the Northwest Regional Education 
Laboratory this system uses a network of evaluators at 
sites across the country, evaluating information which 
i s summarized regional ly . Revi ews are provided to 
MicroSIFT's n;?twork members and are included in the 
Resources in Computer Education (RICE) database. 
MicroSIFT's Evaluator's Guide is available to educa- 
tors who are not members of the formal evaluation 
network. 



EPIE/CU: The Educational Products Information Exchange/Consum- 
ers' Union has formed a network of school districts 
across the country to evaluate software and hardware. 
Evaluators at specific sites are trained to use 
EPIE/CU*s evaluation form, and summaries of the 
reviews are compiled in an analysis manual which is 
available by subscription. 



2. Evaluation strategies which have informal networking procedures 
include: 
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• School Microware Reviews , and 

• Courseware Report Card . These publications have access to evalua- 
tors either in-house or out, who provide reviews of software based 
on certain set criteria. Results are compiled and available by 
subscription. 
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3. Evaluation procedures which were developed by organizations whose 
primary functions are the design and distribution of software for 
educational purposes include; 

• MECC (Minnesota Educational Computer Consortium) 

• SOFTSWAP 

• CONDUIT (publishes evaluations in Pipel ine . available by 
subscription ). 

4. Evaluation forms which are de signed to stand alone , and which are 
meant to be used by individuals or small groups : 

• National Council of Teachers of Mathematics (NCTM) 

• Scholastic, Inc. 

• Electronic Learning magazine (a synthesis of several formats of 
evaluation). 

* * * * 

At this point, turn to Page 11 of the Evaluai.ion of Software: a Guide to 
Guides. Pages 11 through 68 of the Guide to Guides contain sample forms 
ganerated by 10 different software evaluation groups. Each of the different 
forms listed previously is includedin the manual, accompanied by an abstract 
which outlines its potential uses to educators. Participants should refer to 
It is suggested that the participants form small groups in order to discuss 
the relative merits or demerits of the forms. Groups night wish to divide 
according to the four types of forms delineated in this section of notes. 
Another means of grouping, participants for this activity would be by content 
(teaching) or subject (interest) areas. For example, science inbuructors may 
wish to compare opinions with other teachers in a similar field of 
instruction. Grade level taught is another possible category forgrouping. It 
is not necessary to break into small groups, particularly the participants 
number fewer than eight or ten. However, time may often be saved by dividing 
the information to be sifted, and then sharing results in a full group 
session. 
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Participants should refer to Participant Handout //2 A for recording coimionts 
reldtlnij to the forms which might be useful In their situation. Participants 
should identify several different forms for evaluating software as being the 
most appropriate ones for their situation. These forms will be used In the 
following section of the Workshop in hands-on review of educational soft- 
ware. 
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Handout //2-A 

Participant Notes 

EVALUATING THE FOI^MS: COMMENTS 

Record the potential usefulness of the forms: 



• MicroSIFT 



comments : 



EPIE/CU 



codnients; 



School Microware Reviews 



conments: 



Courseware Report Card 



comments; 



MECC 



comments : 
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OW 



I I 1^1' 



0 1 2 3 4 5 6 7 8 



0 1 2 3 4 5 6 7 8 



0 1 2 3 4 5 6 7 8 



J \ L 



0 1 2 3 4 5 6 7 8 



0 1 2 3 4 5 6 7 8 
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CONDUIT 



corrwents: 



NCIM 



cofuments: 



• Scholastic 



corrments: 



• Electronic Learning 



conments: 
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SOFTSWAP low high 



coninonts; 



0 1 2 3 4 5 6 7 8 



0 1 2 3 4 5 6 7 0 



0 1 2 3 4 5 6 7 8 



0 1 2 3 4 5 6 7 8 



J \ I \ I — L 



0 1 2 3 4 5 6 7 8 
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SESSION NOTES 
SECTION 3 
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EVALUATION OF SOFTWARE: APPLICATION 



loGlutlefl 1n Uiu Hx Workshop packet ara QxaniplQS of eclucfitioiuil software pro- 
yPiiiDS that rancje from slnplo remedial through specific content«sk1 1 1 lesions. 
The Intent of Including sample software In the package Is to allow comparison 
of evaluation forms among participants. 

Participants should have by this point chosen at least two of the forms found 
in Guide to Guides as appropriate for their evaluative needs. The group 
leader may wish to have multiple copies of each form duplicated to allow par- 
ticipants to write on the forms during the exercise without damaging original 
copies. Groups of participants may be formed for evaluation practice by sever- 
al methods. If there is a shortage of hardware, groups of no more than three 
may share a machine. When sharing a machine, participants will wish to take 
turns Interacting with the computer keyboard (acting as the student). Another 
means of grouping would be by forms chosen. Participants who have selected 
the same form(s) for practice evaluation may find it less time-consuming and 
mnrp productive to evaluate as a team. 



General guidelines: 

1. It is not necessary to evaluate all of the programs included on the 
di skette enclosed. Participants may chose the programs that are 
applicable to their situations (3 or 4 programs are a good sample). 

2. The programs vary in character. Some are "good" and others not so 
"good, " at first glance. Variations in first impressions often 
depend on the evaluator's background and amount of experience with 
educational computers. The purpose of the exercise is to allow 
experimentation with real situations which could occur during evalua- 
tion sessions. 



3. Run through the entire program at least once so that the format, 
directions, and other characteristics of the program are exposed. Do 
this before beginning the written evaluation. 

4. Complete one evaluation form before beginning on the other. Trying 
O -26- 
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to complete any portion of the forms concurrently will usually be 
confusing. 

NOTE: 

If the sample software included in the packet is not compatible with the brand 
of computer available to the participant, participants can still become 
involved in group evaluation. 

The group leader for the session should collect sample programs from local 
schools or from resource centers at district or regional service centers which 
are compatible with the available computers. Include a range of content and 
ability-requirement levels. The process of evaluation is unfortunately more 
cumbersome when more than one diskette is used. In ordtfr for the process of 
evaluation and comparison to function, each machine must be loaded with the 
same programs individually, with participants sharing the program diskette. 
The trainer may take responsibility for this function or appoint a knowledge- 
able participant to load the machines. Another program may be substituted 
when participants finish with the first. Depending upon time constraints, the 
session should Include at least three or four separate lessons in the evalua- 
tion section. 
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SESSION NOTES 
SECTION 4 
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COMPARISON OF FORMS 
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When all participants have completed evaluating at least three lessons on each 
of the forms selected in Section 2, the whole group should reorganize in order 
to discuss findings. 

Large group discussion should include the following steps: 

• Determine which forms were used. Make a master list. Ideally, at 
least eight out of ten forms will have been used by participants. 

• Using chalkboard or wall charts or an overhead projector with blank 
acetate sheets, list as a heading the title of the form . Each form 
used should have Its own set of comments written in some manner by 
the leader. 

Direct participants to Handout #4. Comments on forms used by the 
group may be recorded on this handout. 

• With the group leader functioning as a facilitator, elicit comments 
from the group regarding the usefulness of the forms they employed in 
eva luati on. 

The trainer may: 

* focus comments by referring to the small -group comments on the 
forms (Participant Handout #2-A). 

* include positive and negative aspects of the evaluation form. 

* discuss one form at a time. 

* find that, ideally, participants will have selected a variety of 
forms, so that the group will receive an overview of a larger 
sanple of their peers' opinions. However, the facilitator may 
find that the group has chosen to employ only three or four 
different forms. This would probably be common in a homogeneous 
group of participants (all math teachers or all second-grade 
teachers, for exanpl-s). 
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* remind participants that this exercise is Intended as a means of 
familiarizing them with evaluation formats. The quality and 
appropriateness of the sample software is not in question ~ it is 
the process of evaluation that is inportant. 



Note: The means of recording group responses to the forms is by using wall 
charts or overhead projector transparancies to make a "master list" of 
comments regarding the uses made of the forms. The yroup leader should 
be responsible for recording (and clarifying) comments on each form. 
Participants should be reminded that the forms used in this particular 
session Should be evaluated in light of the situation: the training 
activity is an artificial setting designed to expose participants to 
various evaluative instruments included in the Guide to Guides . The 
forms chosen and used in these hands-on procedures may not necessarily 
be the "best" or the only ones applicable to software they will be 
evaluating in the future. 
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Handout #4 
Participant Notes 

EVALUATION COMPARISONS: Evaluator's position: 

Grade/level of software: 



Discuss and comment on the usefulness of the forms selected for hands-on 
evaluation activity: 



FORM 1 TITLE: 

PRODUCER: 
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Comments: 

PRO CON 
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FORM 2 TITLE: 

PRODUCER: 



Comment s: 
PRO 



FORM 3 



TITLE: 
PRODUCER 



Cotimerits: 
PRO 



FORM 4 



TITLE: 
PRODUCER 



Comments: 
PRO 
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FORM 5 TITLE: 

PRODUCER: 

Comments: 
PRO 



FORM- 6 TITLE: 

PRODUCER : 

Comments: 
PRO 



FORM 7 TITLE: 

PRODUCER 



Comment s: 
PRO 



SESSION NOTES 
SECTION 5 
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SORWARE REVIEWS AND RESOURCES 



The "Software Reviews" section of the Guide to Guides* manual includes three 
sarrple software reviews. Participants may wish to go through these samples as 
a means of discovering contrasts in technique. An interesting outside task 
might be to obtain several reviews from different sources focusing on the same 
software program. Characteristics of reviews which relate to local needs may 
be noted at that time. 

Commercial software reviews are a good point for beginning the process of 
software selection. They can help educators decide whether potential pu- 
rchases are worth evaluating at a local level by software selection committees 
or individuals. 

* Participants may be directed to pages 69-76 (Guide to Guides) 
Resources ; 

The resource section of the Guide to Guides* represents the most comprehensive 
collection of information sources available at time of publication. Cate- 
gories of resources include books, directories (of software evaluation mate- 
rials and reviews), articles dealing with a wide range of software related 
themes, and clearinghouses and information centers (some related to user- 
groups such as CUE and MECC). Included are names of periodicals and reports 
centered on software reviews, periodicals containing software reviews, and 
conputer-accessible databases containing information on educational software 
and evaluations. In addition there are annotated descriptions and order forms 
for various publications from the Regional Exchange at SEDL and from NEREX. 



* Participants may be referred to page 77-96 (Guide to Guides). 
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MORE SUGGESTIONS FOR THE SESSION LEADER 



So you want to have a workshop . . . here are some ideas which will make the 
process more profitable for the participants and easier for the group leader. 

1) Gather all of the materials needed before the workshop: 

5x Workshop packet (contains % Workshop manual, Guide to Guides 

(SEDL/NEREX) and a software sample disk) 
One copy of Evaluation of Education Software: A Guide to Guides for 

each workshop participant 
Wall charts and markers for recording group discussions (substitutes: 

chalkboard, or overhead projector and markers) 
Pencils and note pads for participants u i^ u 

Microcomputers (ideally, no more than three participants should have to 

work at one maching concurrently) 
Power cords for the machines 



2) Check your conputer hardware for compatibility with the sample software 
included in the packet. The disk in the package is designed for use with 
a TRS-80, Model III Microconputer, and will not operate on another brand 
of computer. 

IF YOU HAVE ANOTHER BRAND OF COMPUTER: 

The group leader will wish to locate examples of educational software 
which represent a range of grade levels, and topic/content areas, as does 
the sanple disk. The leader may check with local schools, regional 
centers, libraries, and vendors in the area who inight be willing to loan 
or donate software for use in the workshop. 

3) Set up microcomputers' for the hands-on session. It would be ideal if each 
participant could have access to his/her own terminal, but this usually is 
not the case. Set up available machines in comfortably small groups (2 or 
3 individuals at each microconputer) and stress that each person in the 
group have an opportunity to interact with the computer at some point 
during the hands-on sessions. 

4) Resources for display may be obtained from various sources: 

• Catalogs: Look for companies which classroom test or teacher-approve 
their materials. The classroom-oriented producers often wil. donate 
copies of catalogs and directories for distribution to workshop parti- 
cipants, 

• Journals/Reviews: Look in the back of the Guide to Guides for some 
suggestions on journals and magazines that address classroom and educa- 
tional microcon^uting practices directly. If possible, obtain some 
current copies of several publications for display during the workshop. 
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Commercial or Publishers' Reviews: Request reprints or photocopy 
published reviews of existing software which might impact the classroom 
situations of the participants. A good tool for practicing the 
evaluation of software is to obta,in a copy of a commercial program or 
package that has been reviewed by another party. In a hands-on 
setting, review the material, and compare results with the published 
review. Perspective may thus be gained by the participants as to how 
accurate and reliable a journal review of educational courseware may be 
in their respective situations. 
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